Investigation of demographic implicit discrimination and disparate impact in chest radiography image-based AI for COVID-19 severity prediction

It is important to evaluate medical imaging artificial intelligence (AI) models for possible implicit discrimination (ability to distinguish between subgroups not related to the specific clinical task of the AI model) and disparate impact (difference in outcome rate between subgroups). We studied potential implicit discrimination and disparate impact of a published deep learning/AI model for the prediction of ICU admission for COVID-19 within 24 hours of imaging. The IRB-approved, HIPAA-compliant dataset contained 8,357 chest radiography exams from February 2020-January 2022 (12% ICU admission within 24 hours) and was separated by patient into training, validation, and test sets (64%, 16%, 20% split). The AI output was evaluated in two demographic categories: sex assigned at birth (subgroups male and female) and self-reported race (subgroups Black/African-American and White). We failed to show statistical evidence that the model could implicitly discriminate between members of subgroups categorized by race based on prediction scores (area under the receiver operating characteristic curve, AUC: median [95% confidence interval, CI]: 0.53 [0.48, 0.57]) but there was some marginal evidence of implicit discrimination between members of subgroups categorized by sex (AUC: 0.54 [0.51, 0.57]). No statistical evidence for disparate impact (DI) was observed between the race subgroups (i.e. the 95% CI of the ratio of the favorable outcome rate between two subgroups included one) for the example operating point of the maximized Youden index but some evidence of disparate impact to the male subgroup based on sex was observed. These results help develop evaluation of implicit discrimination and disparate impact of AI models in the context of decision thresholds © COPYRIGHT SPIE. Downloading of the is permitted for personal use only.

Assistance tools for the evaluation of machine learning algorithm performance: The decision tree-based tools developed by the Medical Imaging and Data Resource Center (MIDRC) Technology Development Project (TDP) 3c effort

MIDRC was created to facilitate machine learning research for tasks including early detection, diagnosis, prognosis, and assessment of treatment response related to the COVID-19 pandemic and beyond. The purpose of the Technology Development Project (TDP) 3c is to create resources to assist researchers in evaluating the performance of their machine learning algorithms. An interactive decision tree has been developed, organized by the type of task that the machine learning algorithm is being trained to perform. The user can select information such as: (a) the type of task, (b) the nature of the reference standard, and (c) the type of the algorithm output. Based on the user responses, they can obtain recommendations regarding appropriate performance evaluation approaches and metrics, including literature references, short video tutorials, and links to available software. Five tasks have been identified for the decision tree: (a) classification, (b) detection/localization, (c) segmentation, (d) time-to-event analysis, and (e) estimation. As an example, the classification branch of the decision tree includes binary and multi-class classification tasks and provides suggestions for methods and metrics as well as software recommendations, and literature references for situations where the algorithm produces either binary or non-binary (e.g., continuous) output and for reference standards with negligible or non-negligible variability and unreliability. The decision tree has been made publicly available on the MIDRC website to assist researchers in conducting task-specific performance evaluations, including classification, detection/localization, segmentation, estimation, and time-to-event tasks. © COPYRIGHT SPIE. Downloading of the is permitted for personal use only.

Longitudinal robustness of a thoracic radiographic AI model for COVID-19 severity prediction

The 'ging' of artificial intelligence/machine learning (AI/ML) models after initial development and evaluation is known to frequently occur and can pose substantial problems. When there are changes in population, disease characteristics, imaging equipment, or protocols, model performance may start to deteriorate, and the performance predicted in a research setting may no longer hold after deployment (either in a clinical setting or in further research). This data shift phenomenon is a common problem in AI/ML. We trained and evaluated a previously in-house developed AI/ML model for COVID severity prediction using two COVID-19-positive consecutive adult patient cohorts from a single institution. The first cohort was from the time that the Delta strain was dominant accounting for <95% of cases (June 24-December 11, 2021, 820 patients, 1331 chest radiographs (CXRs)) and the second cohort was from the time that the Omicron variant was dominant (Jan 1-21, 2022, 656 patients, 970 CXRs). Inclusion criteria were COVID-positivity and the availability of CXR imaging exams, in general for patients not admitted to ICU and prior to ICU admission for those patients admitted to ICU as part of their treatment. Exclusion criteria were image acquisition in ICU or the presence of mechanical ventilation. Our image-based AI/ML model was trained to predict, based on each frontal CXR from a COVID-positive patient, whether this patient would be admitted to ICU within a 24, 48, 72, or 96-hour window. The model was evaluated 1) in a cross-sectional test when trained on a subset/tested on an independent subset of the Delta cohort, 2) similarly for the Omicron cohort, and 3) in a longitudinal test when trained on the Delta cohort/tested on the Omicron cohort. Cohorts were similar in ICU admission rate and fraction of portable CXRs, while immunization rate was higher for the Omicron cohort. The model did not demonstrate signs of aging with performances in the longitudinal test being very similar to those within the Delta cohort, e.g., an area under the ROC curve in the task of predicting ICU admission within 24 hours of 0.76 [0.68;0.84] when trained/tested within the Delta cohort and 0.77 [0.73;0.80] for the longitudinal test (p>0.05). The performance within the Omicron cohort was similar as well, at 0.76 [0.66;0.84]. Our AI/ML model for COVID-severity prediction did not demonstrate signs of aging in a longitudinal test when trained on the Delta cohort and applied as-is to the Omicron cohort. © COPYRIGHT SPIE. Downloading of the is permitted for personal use only.

Sequestration methodology in practice through evaluation of joint demographic distributions of 54,185 patients in the Medical Imaging and Data Resource Center (MIDRC) data commons

The Medical Imaging and Data Resource Center (MIDRC) is a multi-institutional effort to accelerate medical imaging machine intelligence research and create a publicly available data commons as well as a sequestered commons for performance evaluation of algorithms. This work sought to evaluate the currently implemented methodology for apportioning data to the public and sequestered data commons by investigating the resulting distributions of joint demographic characteristics between the public and sequestered commons. 54,185 patients whose de-identified imaging studies and metadata had been submitted to MIDRC were previously separated into public and sequestered commons using a multi-dimensional stratified sampling method, resulting in 41,556 patients (77%) in the public commons and 12,629 patients (23%) in the sequestered commons. To compare the balance obtained in the joint distributions of patient characteristics from use of the developed sequestration method, patients from each commons were separated into bins, representing a unique combination of the demographic variables of COVID-19 status, age, race, and sex assigned at birth. The joint distributions of patients were visualized, and the absolute and percent difference in each bin from an exact 77:23 split of the data were calculated. Results indicated 75.9% of bins obtained differences of less than 15 patients, with a median difference of 3.6 from the total data for both public and sequestered commons. Joint distributions of patient characteristics in both the public and sequestered commons closely matched each other as well as that of the total data, indicating the sequestration by stratified sampling method has operated as intended. © 2023 SPIE.

Sequestration of Imaging Studies in MIDRC: A Multi-Institutional Data Commons

The Medical Imaging and Data Resource Center (MIDRC) is a multi-institutional effort to accelerate medical imaging machine intelligence research and create a publicly available image repository/commons as well as a sequestered database for performance evaluation and benchmarking of algorithms. After de-identification, approximately 80% of the medical images and associated meta-data will become part of the open repository and 20% will be sequestered and kept separate from the open commons. To ensure that both the public, open dataset and the sequestered dataset are representative of the population available, demographic characteristics across the two datasets must be balanced. Our method uses multidimensional stratified sampling where several demographic variables of interest are sequentially used to separate the data into individual strata, each representing a unique combination of variables. Within each stratum, patients are randomly assigned to the open set (80%) or the sequestered set (20%). Thus, for p variables of interest, the balance of the pdimensional distribution of variable combinations can be controlled. This algorithm was used on an example COVID-19 dataset containing image exams of 4662 patients using the variables of race, age, sex at birth, and ethnicity, each containing 8, 8, 2, and 4 categories, respectively. After stratification of this dataset into the two subsets, resulting distributions of each variable matched the distribution from the original dataset with a maximum percent difference from its original fraction of 0.4%. These results demonstrate that the implemented process of multi-dimensional sequential stratified sampling can partition a large database while maintaining balance across several variables. © 2022 SPIE. All rights reserved.